Robot Musical Accompaniment: Real-time Synchronization using Visual Cue Recognition

In a typical musical ensemble performance, each musician must play at the same speed, at the same location in the piece. When this happens, we say they are synchronized. Currently, musical accompaniment systems such as [1] [2] achieve good synchronization with human soloists. However, in larger ensembles with many players playing different parts, synchronization becomes difficult. Several existing approaches in audio processing could be applied for large ensemble synchronization. Score followers such as [3] are capable of dealing with polyphonic input, though have the constraint of requiring prior knowledge of all players’ scores. This is not the case in real musical ensembles; each player typically only knows his own part. Beat trackers such as [4] can also deal with polyphonic input, but a percussive beat is required in order to reliably extract a tempo. At the same time, studies in music [5] suggest that visual cues, such as a conductor’s baton movements or musicians’ body movements, are necessary for temporal coordination. Indeed, certain musical situations, such as a simultaneous first note of a piece, are virtually impossible to synchronize using audio alone. In light of these issues, we propose a vision-based accompanist system which a) starts and stops in synchrony with the lead player b) adapts to tempo changes indicated by the leader c) can be used for polyphonic music or noisy environments and d) by nature, does not require a full score a priori nor use of percussion. As a starting point for future research, we have implemented a filtered Hough line detector and state machine to recognize visual cues (or gestures) of a human flute player. We install the system onto a robot thereminist [6] and synthetic vocalist, which use this visual information to play a piece in synchrony with one human lead player.